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ABSTRACT 

Quality assurance programs have become an integral part of the health delivery 
professions during the past quarter century. They generally focus on the 
appropriateness and adequacy of services provided , with the objective of 
identifying and correcting deficiencies of service. Appropriateness and adequacy 
are assessed by the degree of adherence of practices to explicit standards, and 
these in turn relate to criteria of structure, process, outcome, and efficiency. The 
key concepts and methodologies of quality assurance programs are reviewed, and 
it is suggested that they can be readily applied to Canadian universities. A quality 
assurance program within a university could be a powerful administrative and 
curricular development tool that would serve to define and enhance the quality of 
educational experiences of students. 


RESUME 

Depuis vingt-cinq ans environ les programmes de garantie font partie integrante 
des responsabilites qui incombent aux associations professionnelles medicates. 
En general ces programmes visent d evaluer si les services rendus se sont reveles 
adequats et appropries, et dfaire apparaitre toute carence eventuelle dans le but 
d’y remedier. Ces criteres d’adequation sont evalues enfonction d un principe de 
concordance entre certaines pratiques et certaines normes clairement etablies, 
lesquelles correspondent a des criteres choisis en fonction de la structure du 
programme lui-meme, des moyens d’action utilises, de leurs consequences et de 
leur efficacite. Dans cet article, on passe en revue les methodes employees dans le 
cadre de ces programmes et les concepts ties qui les sous-tendent. On y sugge re dt 
plus que ces programmes pourraient facilement s’appliquer aux universites 
canadiennes. En effet, un programme de garantie au sein de iuniversite pourrait 
etre un outil puissant de perfectionnement tant des structures administrates que 
des programmes eux-memes et it servirait aussi d definir ce que l on entend par e 
concept de qualite dans le domaine de I’enseignement vecu par les etudiants 
universitaires et a se rapprocher toujours davantage de cet ideal. 


7X William G. Webster 


criterion there is a standard against which practices are judged, and this is a 
numerical statement of what constitutes goodness. Standards can be statements 
concerning goodness of optimal care, adequate care, or minimal care; there is no 
agreement as to which type of standard is most appropriate. The use of optimal 
standards leaves room for improvement, but it has the potential of conveying 
inadequacies to those outside the process. On the other hand, use of minimal or 
adequate standards can have the effect of preserving the status quo and militating 
against change and improvement. In summary, quality of service delivery is a 
quantitative statement (e.g., percentage of cases, amount of change, etc.) of the 
extent to which the standard for each criterion is met. 

The key to the development and implementation of a quality assurance program 
is how the criteria for quality are decided upon and how the standards are 
formulated. This has been discussed in detail by Donabedian (1980), Demlo 
(1983), and Williamson (1978). Following a review of the literature, criteria and 
standards are derived normatively through expert opinion solicited, collated, and 
synthesized by panels. For a panel to have the necessary credibility, it must be 
constituted so that the members are knowledgeable of the science and technology 
of medicine, have relevant clinical experience in community and academic- 
settings, have an interest in quality and its assessment, and have certain personal 
characteristics that facilitate effective panel participation. In addition, the panels 
must be representative and balanced with respect to various interest groups. 
Attention has been paid to the appropriate size and, especially, the organization of 
panels. One method used with some success is the Delphi technique (Dalkey et al., 
1972). There are many variations on how the method is actually used but, in 
general terms, its distinctive features are that panel members prepare their answers 
to questions individually, the answers are collated into statistical group responses, 
and members then reconsider their responses in light of the group responses. This 
approach appears to overcome many of the difficulties encountered with 
congregate groups unless those groups operate according to structured procedures 
such as the Nominal Group Technique (Delbecq & Van de Ven, 1971; O’Neil, 
1981). Although there are critics of the Delphi technique (Sackman, 1975), 
Donabedian (1980) suggests that it is most useful when “there is considerable 
uncertainty about the correct answer because arriving at it requires a combination 
of factual knowledge with personal judgment and values” (p. 658), and he presents 
a convincing argument for its success. 

Implementation of quality assurance is carried out on an on-going basis through 
either retrospective review of records or a concurrent review of all patients at a 
given time. Although often there will be an institutional coordinator, the actual 
process of quality assessment is carried out by the practitioners themselves 
exercising whatever judgment is required as to whether process or outcome 
standards have been met. Assessments often are carried out with just one or two 
criteria at a time so that the process does not become overwhelming and 
excessively time consuming. Such a procedure also means that the modification 
of practices remains within manageable proportions when deficiencies are 
encountered. 
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Social scientists will see readily the conceptual and methodological linkages 
that exist between quality assurance in the health domain and program evaluation 
in the public policy and social services domains. Based on a systematic 
comparison of the two. Green and Attkisson (1984) have argued that, despite an 
increasing convergence of the two in recent years, there are significant differences 
in emphasis along several key dimensions which reflect their different histories 
and substantive concerns. A detailed consideration of their analysis is beyond the 
scope of this paper. 

The health delivery system is not dissimilar to that of universities in terms of 
organizational complexity, disciplinary/professional heterogeneity, professional 
autonomy and self-governance, institutional independence, and major public 
involvement in financing. As was previously discussed, health delivery quality 
assurance programs rely on criteria and standards that are developed by the 
individual professions themselves. To the extent that quality assurance programs 
in health delivery have, in fact, been effective in promoting quality in an 
economical manner (Beinecke, 1984), the parallels suggest that there could be 
value in universities adopting quality assurance concepts and methodologies for 
promoting the quality of their own activities. Programs that stress self- and 
peer-review and that have the objective of enhancing and optimizing the 
appropriateness and adequacy of service delivery to individuals are compatible in 
principle with university traditions and values. How the concepts and methodolo¬ 
gies of quality assurance might be adopted will be considered specifically in the 
next section. 


EVALUATION IN HIGHER EDUCATION 

A substantial body of literature now exists concerned with the impact of higher 
education on students (Bowen, 1977; Pace, 1979; Halpern, 1987; Conrad & 
Wilson, 1985; Winter, McClelland & Stewart, 1981). Within the literature, the 
predominant focus of attention has been on outcome variables such as knowledge, 
skills, cognitive development, attitudes and values. For example. Winter et al. 
(1981) developed measures of cognitive, motivational and emotional outcomes to 
test their hypothesis that the effects of a liberal arts education are largely operant 
behaviours, specifically “processes for operating on and using information rather 
than mere knowledge of facts or information.” Similarly in studies that have 
assessed the quality of individual programs of study or individual institutions 
(e.g., Krueger & Heisserer, 1987; Curry & Hagar, 1987; Ciereszko, 1987; 
Osigweh, 1988), measures have generally been derived from standardized tests of 
knowledge and academic achievement. This literature has been reviewed in some 
detail by Conrad & Wilson (1985) and, as a broad generalization, most studies of 
higher education have focused on outcome and have had the objective of 
demonstrating a positive effect of the college experience (often the liberal arts 
experience) on the individual student. 

The approach to evaluation of higher education that appears to come closest 
conceptually to quality assurance, as developed in this paper, is what has been 
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referred to as “value-added” (Osigweh, 1986, 1988) or, as preferred by Astin 
(1987), “talent development.” Astin makes the very useful distinction between 
outcome and value-added. Outcome refers to the level of developed competency 
or talent of graduates regardless of how it might have been acquired or developed. 
Clearly, it confounds student selection with experiential factors. Value-added, on 
the other hand, is a concept derived from economics and stresses the contribution 
made by the institution and the educational experience of the student in the 
development of the competency or talent. While a focus on change removes in part 
the influence of student selection factors, the basic problem with stressing global 
outcome still remains: In a quality assurance context many outcomes are beyond 
the direct control of the institution. Equally important, the origin of the particular 
outcome or change is difficult to identify and, hence, modify. This is exactly the 
same case as with the health delivery system, and it underlies the widespread use of 
process criteria even though their link to outcome may be only presumptive. 

Although the actual relationship between process and outcome criteria is usually 
not known, this should not unduly delay implementation of a quality assurance 
program. In the context of course and curriculum planning, assumptions are 
already made about the relationship, and they can continue to be made but need to 
be systematically evaluated whenever possible. Clearly, an effective quality 
assurance program must include a component of continuing research to validate 
process criteria and presumed linkages of these criteria to specific outcomes. This 
cannot be overemphasized. 

IMPLEMENTING QUALITY ASSURANCE 
IN A FACULTY OF SOCIAL SCIENCES 2 

Experience in the health delivery field suggests a number of principles that may be 
important for any effective system of quality assurance: 

1. The develoment of criteria and standards must be carried out by those 
charged with service delivery, i.c., the faculty, with broad consultation among 
interested constituencies within both the university and broader community. 

2. The evaluation of quality, i.e., the assessment of the extent to which 
standards are met in practice, must be carried out by faculty themselves. 
Establishment of a separate bureaucracy to provide coordination, guidance, and 
continuity may be desirable and necessary, but the role of the bureaucracy cannot 
be to establish standards by itself or to evaluate quality. 

3. Whatever system is used must be simple to implement by those who are not 
specifically trained in evaluation activities and for whom evaluation activities may 
be of little inherent interest. 

4. As emphasized by Madaus and Stufflebeam (1984), quality assurance 
entails on-going study of activities aimed at improvement. Accordingly, the 
effective development and implementation of a program requires understanding, 
commitment, and support from the most senior levels of the institution (Kirkwood, 
1985). While necessary, this is not sufficient. Halpern (1987) has reminded us that 
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top-down pressure often meet skepticism and resistance within a university, and 
consequently faculty support for all aspects of a quality assurance program is 
critical for its success. 

5. The focus of evaluation should not be the individual instructor nor the 
individual course; it is the degree program and the experiences of individual 
students in following their degree program. As ironic as it may seem, a quality 
assurance program may not intersect significantly with the processes of teaching 
evaluation or personnel appraisal, although it can have direct implications for 
course requirements and pedagogy. 

A useful system of quality assurance that is consistent with these general 
principles could be developed and implemented within a Faculty of Social 
Sciences by addressing systematically three broad questions. 

1. What are the objectives or desired outcomes of a social sciences under¬ 
graduate degree program? Can these objectives be expressed in terms of skills, 
talents, attributes, qualities, or competencies that graduates should have devel¬ 
oped by virtue of following and completing the degree program? 

Drawing upon health delivery, this question could be addressed through a panel 
with representatives from different disciplines within Social Sciences and 
operating according to principles of the Delphi or Nominal Group technique. The 
multitude of ideas in the literature as to what ought to constitute a liberal arts 
education (e.g., see reviews by Bloom, 1956; Donald, 1985;Ewell, 1987;Winter, 
McClelland and Stewart, 1981) can be drawn upon in the consideration and 
articulation of generic objectives. 

Concurrently, other panels with representatives from within each different 
social science discipline would be attempting to develop discipline-specific 
objectives and criteria. These might include a list of ideas or concepts within the 
major about which a graduating student would be expected to have a certain level 
of familiarity and understanding. One might expect such ideas to be embodied 
implicitly in the curriculum already and, accordingly, their articulation would not 
require a major time commitment from panel members involved. 

2. Once sets of generic and discipline-specific goals and outcomes are agreed 
to, 3 the second issue becomes the specification of the processes or the experiences 
thought to be sufficient and/or necessary for each outcome to become developed. 
Note that the emphasis is on opportunity rather than achievement because it is 
opportunity (process) rather than achievement (outcome) over which an institution 
principally has control. It is on this point of stressing process that this particular 
quality assurance approach to higher education departs significantly from others, 
including value-added. 

By way of example of this second step, if confidence and competence in oral 
communication were a desired outcome, the next question becomes, “What 
experiences must the student have had at some point in the degree program in order 
to have had the opportunity to develop the skill to a desired level?” Presumably, 
one might include a requirement to make oral presentations in courses, and one 
could define and describe just what constitutes an oral presentation. In the 
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terminology of quality assurance, this would be a criterion. Then one must specify 
the number of such presentations that would constitute an ideal, adequate, or 
minimal standard. Hence, one might set as an adequate standard that the student is 
required “to have made in classes or seminars at least four oral presentations of at 
least 5 minutes each and of which at least one involved defending an idea or 
position.” In the case of many specific skills, these criteria and standards may be 
relatively easy to develop in quantitative terms. In the case of intellectual skills like 
analytic thought, problem solving, or applications of information, greater reliance 
may have to be placed on indirect indicators and acceptance of multiple equivalent 
experiences will be required. Regardless of ease of definition and specification, it 
is important to remember as well that the process of discussing objectives and 
experiences raises sensitivity to quality and so by itself can be an important 
component of improving quality. 

3. The third phase is to proceed to assess quality by determining the extent to 
which standards of particular criteria are met in the experiences of individual 
students. To reiterate, quality is a quantitative statement (e.g., percentage of 
cases, amount of change) of the congruence between standards and practices. We 
might, for example, have a standard of “having written at least 3 essays, at least 
one of which requires a library search of literature and the development of a 
hypothesis, and at least two of which are returned to students with specific 
comments on grammar and organization as well as substance.” Wc can then ask, 
“In what proportion of graduating students in the degree program has this standard 
in fact been met?” To answer this might require determining individual course 
requirements over a period of years, developing a data base, and then tracking 
students through their programs. An alternative and complementary approach (of 
equally unknown reliability) would be to send questionnaires to a sample of 
graduating students asking about their courses and what was required in them with 
respect to certain specific criteria being examined. 

A number of points are to be kept in mind when considering implementation. 
First, a program does not attempt to evaluate quality with respect to all criteria at 
one time. Instead, the focus of the exercise will be one subset of criteria at one time 
and another subset at another time. Some criteria may be judged to be of greater 
relative importance and they would be monitored regularly. This feature keeps 
the process within manageable proportions. It also permits experience to be gained 
with the process before requiring a major commitment to data collection. 

Second, assessment of quality is not the end in itself. Instead, it leads to a 
consideration of means to improve quality. To the extent that standards are not 
being met within some realm, a department would be expected to review and 
modify the organization and pedagogy of its undergraduate program so that the 
standards are met for an increased proportion of students. This might entail an 
agreement as to which courses would have requirements that relate to specific 
standards, for example, which courses would have an essay assignment 
requirement. If the program exceeds the minimal or adequate standard with respect 
to one criterion, then the question becomes whether the program might be changed 
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so that support resources could be diverted to help the department meet the 
standard with respect to other criteria. 

A third point is that the whole process of'quality assurance approached from the 
perspective of the extent to which standards are met hinges on definitions of 
standards. This leads to a consideration of two issues. The first is that, in the 
development ol criteria and standards, ideal, adequate, and minimal standards 
must be distinguished. We should not confuse these. We do not expect to meet the 
ideal; we expect, however, to consistently exceed the minimum. My assumption is 
that it would be far easier to achieve consensus with respect to what would 
constitute adequate or minimal standards than optimal standards, in part because it 
allows one to skirt around the argument that many of the ideals of higher education 
are in some sense ethereal and elude measurement. Of potential significance is the 
impression, based on discussions with individuals in the health delivery field, that 
adequate standards are easiest to wok with psychologically in that much of the time 
one would expect they would be met or exceeded but they still provide room for 
improvement. The second issue with respect to criteria and standards is that quality 
assurance is only as good as the criteria and standards used. Accordingly, some 
external review process may be beneficial. While external consultants cannot 
specify what the criteria and standards should be, they can usefully comment on 
what they might be in light of practices elsewhere. In this respect, the process is 
similar to reviews of graduate programs carried out by bodies such as the Ontario 
Council on Graduate Studies. 

A CONCLUDING OBSERVATION 

Some of the concepts and methods of quality assurance in the health delivery field 
have been reviewed and I have suggested that they can be applied to Canadian 
higher education. Although there are many obvious political and public relations 
reasons for universities to have in place an objective and documentable account¬ 
ability system concerned with quality and cost-effectiveness, the principal 
rationale for such a program remains the enhancement of the quality of the 
educational experience of individual students. Furthermore, the accountability 
process itself, when approached from a quality assurance perspective and imple¬ 
mented in an on-going manner, is one that raises consciousness about educational 
objectives and the means by which such objectives are met. This process also 
points to areas that require improvement, and supports departments and individual 
faculty members in implementing their resolve and dedication to provide the best 
possible educational experience for their students. 

NOTES 

1 This article is based on Report 1 of the Task Force on Quality Indicators, Faculty of Social Sciences, 
Carleton University. 

2 Although this article is written from the perspective of the social sciences, the ideas arc equally 
applicable to any segment of the university. 
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3 The author has recently conducted in-depth interviews with individual faculty members from a 
variety of social science disciplines. The results indicate that, at least within the Faculty with which 
the author is associated, there is remarkable concordance among faculty members across disciplines 
in ideas on generic objectives. This provides some grounds for optimism that reaching agreement on 
objectives or goals may not be as daunting a task as it may first appear. Although Tuckman and 
Chang (1988) have pointed out that differences may be expected to emerge more when one begins to 
operationalize what the goals actually mean, the interviews also indicate concordance in general 
terms with respect to the kinds of experiences thought to be necessary for students to have an 
opportunity to achieve generic objectives. 
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